[api] delete file #1122

MalinAhlberg · 2024-11-07T15:38:50Z

Related issue(s) and PR(s)
This PR closes #1134 .

Description
This PR adds the delete file functionality to the api component. Specifically, it deletes the file from the inbox and it adds a new file log event, setting the file status to disabled.

Also, it adds the fileID to the list functionality of the api, since that field is needed in order to delete a file:

curl -H "Authorization: Bearer $token" "http://localhost:8090/users/[email protected]/files" | jq .
[
  {
    "fileID": "d32f7117-bb85-40e2-9c52-101bf9c1ca5a",
    "inboxPath": "test_dummy.org/race_file.c4gh",
    "fileStatus": "ready",
    "createAt": "2024-11-19T10:50:07.836513Z"
  },
  {...

How to test
make build-all then PR_NUMBER=$(date +%F) docker compose -f .github/integration/sda-s3-integration.yml run integration_test.
List the files (eg with http://localhost:8090/users/[email protected]/files) and make sure files in the inbox can be deleted, and that archived files can not be deleted.

kostas-kou

Good job. I have only some minor comments

sda/cmd/api/api.go

sda/internal/database/database.go

.github/integration/tests/sda/60_api_admin_test.sh

sda/cmd/api/api.md

MalinAhlberg · 2024-11-27T08:17:32Z

Great comments @kostas-kou ! Fixed most of them in 92903a6, ~~but left this one for others to see~~ <- solved during stand-up.

pahatz

Great work! Especially on the tests, very extensive.
I don't have any significant remark on the PR.

sda/cmd/api/api.go

sda/internal/database/db_functions.go

MalinAhlberg · 2024-11-29T08:54:10Z

@kostas-kou and @pahatz, thanks for your reviews! I have fixed the weird comment, rebased on main and also rebased to get rid of the fixup-commits. Only 3ac7e5e is new, the rest is the same as when you reviewed.

MalinAhlberg · 2024-11-29T09:39:59Z

...and added 8a745c4 for the rbac

aaperis

Haven't tested, but it looks good. Tiny minor comments :-)

.github/integration/tests/sda/60_api_admin_test.sh

sda/cmd/api/api.md

sda/cmd/api/api.go

jbygdell

More specific comments are coming

jbygdell

There is a few things here that we need to think more about.

In FEGA the submitter can trigger ingestion of an uploaded file. That means we also needs to be able to remove a file both from the inbox and the archive during a cleaning operation. As well as the backup site if that is configured.
The API can also be used by the submitter and that means we can't use the same functions or structs sometimes as that might leak things the submitter should not be able to see.

.github/integration/sda/rbac.json

sda/cmd/api/api.go

.github/integration/sda/rbac.json

sda/cmd/api/api.go

.github/integration/tests/sda/60_api_admin_test.sh

sda/internal/database/database.go

sda/internal/database/db_functions.go

MalinAhlberg · 2024-12-05T08:05:33Z

Thanks @jbygdell for the comments! I will fix the trivial ones, and have answered to some. The more complex ones (these) should be dicussed in the team/with the PO.

sda/cmd/api/api.md

sda/cmd/api/api.go

Co-authored-by: Joakim Bygdell <[email protected]>

MalinAhlberg · 2024-12-05T14:41:30Z

Updated, fixed, rebased etc. Please re-review.

jbygdell · 2024-12-05T15:16:15Z

.github/integration/tests/sda/60_api_admin_test.sh

+fi
+
+# Try to delete an unknown file
+resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/badfileid")"


Suggested change

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/badfileid")"

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -X DELETE "http://api:8080/file/[email protected]/badfileid")"

jbygdell · 2024-12-05T15:16:30Z

.github/integration/tests/sda/60_api_admin_test.sh

+
+# Try to delete file of other user
+fileid="$(curl -k -L -H "Authorization: Bearer $token" -H "Content-Type: application/json" "http://api:8080/users/[email protected]/files" | jq -r '.[0]| .fileID')"
+resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/$fileid")"


Suggested change

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/$fileid")"

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -X DELETE "http://api:8080/file/[email protected]/$fileid")"

jbygdell · 2024-12-05T15:17:46Z

.github/integration/tests/sda/60_api_admin_test.sh

+    exit 1
+fi
+
+fileid="$(curl -k -L -H "Authorization: Bearer $token" -H "Content-Type: application/json" "http://api:8080/users/[email protected]/files" | jq -r '.[] | select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") | .fileID')"


Suggested change

fileid="$(curl -k -L -H "Authorization: Bearer $token" -H "Content-Type: application/json" "http://api:8080/users/[email protected]/files" | jq -r '.[] | select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") | .fileID')"

fileid="$(curl -k -L -H "Authorization: Bearer $token" "http://api:8080/users/[email protected]/files" | jq -r '.[] | select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") | .fileID')"

jbygdell · 2024-12-05T15:21:39Z

.github/integration/tests/sda/60_api_admin_test.sh

+until [ "$(psql -U postgres -h postgres -d sda -At -c "select id from sda.file_events e where e.title in (select event from sda.file_event_log where file_id = '$fileid' order by started_at desc limit 1);")"  -gt 30 ]; do
+   echo "waiting for ingest to complete"
+   RETRY_TIMES=$((RETRY_TIMES + 1))
+   if [ "$RETRY_TIMES" -eq 30 ]; then
+      echo "::error::Time out while waiting for upload to complete"
+      exit 1
+   fi
+   sleep 2
+done


Since ingest automatically triggers a verification event why not use that as a watcher, like we do in 20_ingest-verify_test

echo "waiting for verify to complete" RETRY_TIMES=0 until [ "$(curl -su guest:guest http://rabbitmq:15672/api/queues/sda/verified/ | jq -r '.messages_ready')" -eq 4 ]; do echo "waiting for verify to complete" RETRY_TIMES=$((RETRY_TIMES + 1)) if [ "$RETRY_TIMES" -eq 30 ]; then echo "::error::Time out while waiting for verify to complete" exit 1 fi sleep 2 done

jbygdell · 2024-12-05T15:21:57Z

.github/integration/tests/sda/60_api_admin_test.sh

+
+# Try to delete file not in inbox
+fileid="$(curl -k -L -H "Authorization: Bearer $token" -H "Content-Type: application/json" "http://api:8080/users/[email protected]/files" | jq -r '.[] | select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") | .fileID')"
+resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/$fileid")"


Suggested change

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/$fileid")"

resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -X DELETE "http://api:8080/file/[email protected]/$fileid")"

jbygdell · 2024-12-05T15:45:04Z

sda/cmd/api/api.go

+	fileID = strings.TrimPrefix(fileID, "/")
+	log.Debug("submission file:", fileID)
+	if fileID == "" {
+		c.AbortWithStatusJSON(http.StatusBadRequest, "file ID is required")


Suggested change

c.AbortWithStatusJSON(http.StatusBadRequest, "file ID is required")

c.AbortWithStatusJSON(http.StatusBadRequest, "file ID is required")

return

jbygdell · 2024-12-05T15:46:19Z

sda/cmd/api/api.go

+	submissionUser := c.Param("username")
+	log.Debug("submission user:", submissionUser)
+
+	fileID := c.Param("file")


Suggested change

fileID := c.Param("file")

fileID := c.Param("fileid")

jbygdell · 2024-12-05T15:46:35Z

sda/cmd/api/api.go

+
+	fileID := c.Param("file")
+	fileID = strings.TrimPrefix(fileID, "/")
+	log.Debug("submission file:", fileID)


Suggested change

log.Debug("submission file:", fileID)

log.Debug("submission fileID:", fileID)

jbygdell · 2024-12-05T17:39:52Z

sda/cmd/api/api.go

+	var RetryTimes = 5
+	for count := 1; count <= RetryTimes; count++ {
+		log.Warn("trying to remove file from inbox, try", count)
+		err = inbox.RemoveFile(filePath)
+		if err != nil {
+			log.Errorf("Remove file from inbox failed, reason: %v", err)
+		}
+
+		// The GetFileSize returns zero in case of error after retrying a number of times
+		fileSize, _ := inbox.GetFileSize(filePath)
+		if fileSize == 0 {
+			break
+		}
+
+		time.Sleep(time.Duration(math.Pow(2, float64(count))) * time.Second)
+	}


Suggested change

var RetryTimes = 5

for count := 1; count <= RetryTimes; count++ {

log.Warn("trying to remove file from inbox, try", count)

err = inbox.RemoveFile(filePath)

if err != nil {

log.Errorf("Remove file from inbox failed, reason: %v", err)

}

// The GetFileSize returns zero in case of error after retrying a number of times

fileSize, _ := inbox.GetFileSize(filePath)

if fileSize == 0 {

break

}

time.Sleep(time.Duration(math.Pow(2, float64(count))) * time.Second)

}

for count := 1; count <= 5; count++ {

err := inbox.RemoveFile(filePath)

if err == nil {

break

}

if count == 5 {

log.Errorf("remove file from inbox failed, reason: %v", err)

c.AbortWithStatusJSON(http.StatusInternalServerError, ("remove file from inbox failed")

return

}

time.Sleep(time.Duration(math.Pow(2, float64(count))) * time.Second)

}

This is but one way of dealing with a retry functionality.
It's also possible to have the err in the check part of the for loop like this:

for count := 1; (err != nil && count <= 5); count++ { err = inbox.RemoveFile(filePath) time.Sleep(time.Duration(math.Pow(2, float64(count))) * time.Second) } if err != nil { log.Errorf("remove file from inbox failed, reason: %v", err) c.AbortWithStatusJSON(http.StatusInternalServerError, ("remove file from inbox failed") return }

sda/cmd/api/api.go

dbampalikis assigned MalinAhlberg, nanjiangshu and dbampalikis Nov 7, 2024

MalinAhlberg force-pushed the feature/api-delete-file branch 8 times, most recently from b31bc38 to 446a1ff Compare November 21, 2024 10:22

MalinAhlberg marked this pull request as ready for review November 21, 2024 10:22

MalinAhlberg requested a review from a team November 21, 2024 10:25

kostas-kou reviewed Nov 26, 2024

View reviewed changes

sda/cmd/api/api.go Outdated Show resolved Hide resolved

sda/internal/database/database.go Show resolved Hide resolved

.github/integration/tests/sda/60_api_admin_test.sh Outdated Show resolved Hide resolved

sda/cmd/api/api.md Show resolved Hide resolved

kostas-kou previously approved these changes Nov 28, 2024

View reviewed changes

pahatz previously approved these changes Nov 28, 2024

View reviewed changes

sda/cmd/api/api.go Outdated Show resolved Hide resolved

sda/internal/database/db_functions.go Show resolved Hide resolved

MalinAhlberg dismissed stale reviews from pahatz and kostas-kou via 353261a November 29, 2024 07:52

MalinAhlberg force-pushed the feature/api-delete-file branch from 353261a to 3ac7e5e Compare November 29, 2024 08:51

pahatz previously approved these changes Nov 29, 2024

View reviewed changes

aaperis previously approved these changes Dec 2, 2024

View reviewed changes

.github/integration/tests/sda/60_api_admin_test.sh Show resolved Hide resolved

sda/cmd/api/api.md Outdated Show resolved Hide resolved

sda/cmd/api/api.go Outdated Show resolved Hide resolved

sda/cmd/api/api.go Outdated Show resolved Hide resolved

MalinAhlberg dismissed stale reviews from aaperis and pahatz via 0f445c8 December 2, 2024 13:57

MalinAhlberg force-pushed the feature/api-delete-file branch from 8a745c4 to 0f445c8 Compare December 2, 2024 13:57

jbygdell requested changes Dec 2, 2024

View reviewed changes

MalinAhlberg force-pushed the feature/api-delete-file branch 2 times, most recently from 109e7e2 to d690e54 Compare December 3, 2024 07:43

jbygdell requested changes Dec 3, 2024

View reviewed changes

jbygdell reviewed Dec 5, 2024

View reviewed changes

sda/cmd/api/api.md Outdated Show resolved Hide resolved

MalinAhlberg commented Dec 5, 2024

View reviewed changes

sda/cmd/api/api.go Outdated Show resolved Hide resolved

dbampalikis and others added 20 commits December 5, 2024 15:10

[postgres] Add api user priviledges

cef2c04

[api] Add api configuration

25b456a

[api] Add fileID to api struct

8550847

[api] Change list to return id and skip disabled

222fcb6

[api] Add disable file event log function

c27e85e

[api] Add delete file functionality

3980efd

[api] add integration test

ebc071b

[api delete] only search for unarchived files

6ff7504

[tests] api admin tests

89422c6

[api delete] only search for unarchived files

6f00fc6

[tests] unit test for db function

934bb13

[api] allow deletion of files marked with error

605d9c5

[charts] add inbox conf to api

b455a6f

[api] add documentation

f85e9a9

[api] typos etc

91ded88

[api] rephrase comment

cfcfc27

[tests] add rbac permissions for file deletion endpoint

e93b9cc

Apply suggestions from code review

bc41723

Apply suggestions from code review

31b60b2

Co-authored-by: Joakim Bygdell <[email protected]>

[api] only allow deletion of files marked as uploaded

50367ef

MalinAhlberg force-pushed the feature/api-delete-file branch from 0fa47f5 to 50367ef Compare December 5, 2024 14:30

jbygdell requested changes Dec 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[api] delete file #1122

[api] delete file #1122

MalinAhlberg commented Nov 7, 2024 •

edited

Loading

kostas-kou left a comment

MalinAhlberg commented Nov 27, 2024 •

edited

Loading

pahatz left a comment

MalinAhlberg commented Nov 29, 2024

MalinAhlberg commented Nov 29, 2024

aaperis left a comment

jbygdell left a comment

jbygdell left a comment

MalinAhlberg commented Dec 5, 2024

MalinAhlberg commented Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024

jbygdell Dec 5, 2024 •

edited

Loading

	resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -H "Content-Type: application/json" -X DELETE "http://api:8080/file/[email protected]/badfileid")"
	resp="$(curl -s -k -L -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $token" -X DELETE "http://api:8080/file/[email protected]/badfileid")"

	fileid="$(curl -k -L -H "Authorization: Bearer $token" -H "Content-Type: application/json" "http://api:8080/users/[email protected]/files" \| jq -r '.[] \| select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") \| .fileID')"
	fileid="$(curl -k -L -H "Authorization: Bearer $token" "http://api:8080/users/[email protected]/files" \| jq -r '.[] \| select(.inboxPath == "test_dummy.org/NE12878.bam.c4gh") \| .fileID')"

	log.Debug("submission file:", fileID)
	log.Debug("submission fileID:", fileID)

[api] delete file #1122

Are you sure you want to change the base?

[api] delete file #1122

Conversation

MalinAhlberg commented Nov 7, 2024 • edited Loading

kostas-kou left a comment

Choose a reason for hiding this comment

MalinAhlberg commented Nov 27, 2024 • edited Loading

pahatz left a comment

Choose a reason for hiding this comment

MalinAhlberg commented Nov 29, 2024

MalinAhlberg commented Nov 29, 2024

aaperis left a comment

Choose a reason for hiding this comment

jbygdell left a comment

Choose a reason for hiding this comment

jbygdell left a comment

Choose a reason for hiding this comment

MalinAhlberg commented Dec 5, 2024

MalinAhlberg commented Dec 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbygdell Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

MalinAhlberg commented Nov 7, 2024 •

edited

Loading

MalinAhlberg commented Nov 27, 2024 •

edited

Loading

jbygdell Dec 5, 2024 •

edited

Loading